Near Linear Time Detection of Distance-Based Outliers and Applications to Security

Many automated systems for detecting threats are based on matching a new database record to known attack types. However, this approach can only spot known threats and thus researchers have also be­ gun to use unsupervised approaches based on de­ tecting outliers or anomalous examples. A popular method of finding these outliers is to use the dis­ tance to an example’s k nearest neighbors as a mea­ sure of unusualness. However, existing algorithms for finding distance-based outliers have poor scaling properties, making it difficult to apply them to large datasets typically available in security domains. In this paper, we propose modifications to a simple, but quadratic, algorithm for finding distance-based out­ liers, and show that it achieves near linear time scal­ ing allowing it to be applied to real data sets with millions of examples and many features.